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ABSTRACT 

After more than 10 years into the charter-school movement, a 
fundamental issue remains to be addressed, according to this paper. The issue 
reduces the charter movement down to its basic core premise, which is 
effectiveness. The question for all stakeholders is how effective is the 
charter-school model . Evaluation within the context of the charter-school 
movement can be understood to mean the systematic assessment of the charter- 
school program for the purpose of school improvement. The paper claims that 
the charter-school evaluation process remains under construction with plenty 
of work remaining to be done. The paper presents a charter-school evaluation 
plan with that issue in mind. The plan reflects current best practices of 
charter-school evaluation. The paper focuses on three areas: (1) the 

evaluation criteria of the charter school under examination; (2) the 
relationship of the evaluation process to charter-school effectiveness; and 
(3) the problems and promises of the charter-school plan. The paper concludes 
that evaluation should be both formative and summative in scope. The 
evaluation approach should be manageable and reasonable in dimension and 
intent, and should address the relational and quantifiable problems. (WFA) 
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Introduction 

Starting with Minnesota in 1992 and continuing on to Arkansas in 1999, the 
number of charter schools has continued to grow to a record number of more than 1,400 
in thirty-two states . and the District of Columbia (Bruno, Finn, Bierlein, & Vanourek, 

1998; United States Department of Education, 2000). A total of thirty-six states and the 
District of Columbia have charter school legislation in place. Collectively, charter 
schools are estimated to have an enrollment in excess of a quarter million student with 
the majority of them enrolled in Arizona, California, and Michigan (United States 
Department of Education, 2000). Even the most pessimistic projections suggest that the 
number of charter schools will continue to increase as the demand for school 
improvement and accountability remain a priority issue in public education (Rose, 1999). 

The charter school laws in the various states are based on the proposition that 
allowing schools autonomy from existing bureaucratic and regulatory constraints will 
enable them to adopt innovative and productive methods of operation, finance, and 
governance to support the improvement of teaching and learning (Education Commission 
of the States, 1995). In addition, proponents argue that charter schools will increase the 
number of educational options within the public school system. Most importantly, charter 
schools represent a shift from rule-based accountability to performance-based 
accountability (Bierlein, 1996). In exchange for freedom from bureaucracy, schools agree 
to be held accountable for student performance. As such, they offer one model for 
fundamental reform of the system of schooling (Hill, 1996). 

Nine years into the charter school movement, a fundamental issue remains to be 
addressed. The issue reduces the charter movement to down to its basic core premise 
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which is effectiveness (Joint Committee on Standards for Educational Evaluation, 1981). 
The question for all stakeholders is; How effective is the charter school model? This 
question seems to have multiple responses ranging from poor, fair, great— depending on 
whom you ask and your yardstick. The question, however, generates a myriad of other 
cmcial questions about charter school effectiveness. All of the questions illustrate the 
difficult in determining and measuring effectiveness. For instance: Can effectiveness be 
defined as filled classrooms? Can the number of charter schools in existence define 
effectiveness? Can effectiveness be defined by student achievement? Can effectiveness 
be defined as accessibility for all groups? Can effectiveness be defined as having a 
positive impact on other public schools? Numerous other measures indicating program 
effectiveness could also be asked. Unfortunately, the responses may vary and further 
complicate the issue of charter school effectiveness. 

Singularly these questions will not be adequate in addressing the issue of school 
effectiveness because the issue is extremely complex in the same manner that it is for 
traditional schools. At the same time, collectively these questions may or may not be 
adequate in addressing the charter effectiveness issue because the current evaluation 
approach is too wide and diverse (See Table One). Nationally, the majority of charter 
schools use at least five or more assessment methods of which more than one-third use 
seven assessment methods (United States Department of Education, 2000). Furthermore, 
the current approach rest on the assumption that the seven more common assessment 
methods are relational in nature and that the relationship between all of the methods can 
be quantified. In addition, it is assumed that other assessment methods being used, by 
choice or are tlirough mandate, are compatible with the seven more common assessment 
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methods and that they are also relational and quantifiable. At best, the relationship 
between some of the assessment methods is murky, therefore, so is the relationship 
between them (Worthen & Sanders, 1987). 

Evaluation within the context of the charter school movement can be understood 
to mean the systematic assessment of the charter school program for the purpose of 
school improvement. Nationally, the overwhelming majority of charter schools are using 
multiple assessment methods including standardized assessment through norih referenced 
and criterion referenced assessments, performance assessments, student portfolios, 
student demonstrations, parent surveys, student interviews or surveys, and behavioral 
indicators to determine charter school success (United States Department of Education, 
2000). In addition, some charter schools are assessed on how well they use their financial 
resources, comply with educational regulations, enhance student attendance, manage 
student discipline, and implement best instructional practices. Still others are being 
required to ascertain the instructional, administrative and district related cost associated 
with charter schools (See Table Two). In essence, the charter school evaluation process 
remains under construction with plenty of a work remaining to be done. 

A charter school evaluation plan will be presented with that issue in mind. The 
plan reflects current best practices of charter school evaluation. The researchers will 
focus the discussion on three areas. First, the evaluation criteria of the charter school 
under examination will be discussed. Second, the relationship of the evaluation process to 
charter school effectiveness will be discussed. Summary comments will be offered about 
what happens when the evaluation process is expected to contain all the gifts in Pandora’s 
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Box and the implications this has when the focus is on the gifts and not the meaning of 
those gifts. 

An Elementary Charter School Evaluation Plan 
The evaluation plan presented is the actual plan for an elementary charter school 
in a southern state in the United States. The school has come into existence this year and 
is serving about a hundred at-risk students. The goals of the charter school are to provide: 
(a) an effective early intervention/prevention for academic underachievement, (b) an 
effective early intervention/prevention for socio-behavioral and personal failure, and (c) 
an aggressive approach to rebuilding community and parent identity and support for 
public education through an innovative national model. 

The evaluation approach Used by the school is to assist them in determining 
program effectiveness. The effectiveness of the charter school will be determined by 
whether or not at-riSk 3rd, 4'*’, and 5* grade students, enrolled in the charter school, learn 
and perfomi significantly better than comparable students not in the charter school on a 
variety of achievement and other schooling variables (i.e., grades, attendance, discipline, 
constituent satisfaction, instructional strategies, charter school impact, and cost of 
instruction, administration and transportation). In addition, students leaving the charter 
school will be tracked and achievement as well as other data will be collected and 
evaluated. The achievement of those students will be compared. to students attending the 
charter school. 

The evaluation approach is considered to be comprehensive and the assessment 
methods appropriate to determine charter school effectiveness. The evaluation approach 
with its numerous assessment methods will be discussed (See Table Three). 
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Baseline Data 

To facilitate the evaluation process, baseline data on Rausch (RIT) growth data 
for students in grades 3'^'* through 5'^ will be collected along with data from other tests, 
however, since baseline data is not available for the other tests, descriptive statistics will 
be reported for student scores on the state benchmark exam for 4* grade, the Stanford 
Achievement Test (SAT-9) exam for 5* grade, and the school district’s 2"''.and 3'^'* 
quarter criterion referenced tests. 

In addition, grades, attendance, and discipline data, where appropriate, will be collected. 
Academic Achievement Data 

Academic achievement will be measured using several different tests. First, 
Rausch (RTT) growth from the Northwest Evaluation Association (NWEA) Level Test 
for grades 3'^'* through . Expected growth will be derived from a national comparison 
group of 500,000 students taking the NWEA level tests. The expected growth rate is an 
increase of 4% in reading, 5 % in math, and 3% in language; second, the state benchmark 
exam for 4* grade; third, the Stanford Achievement Test (SAT-9) exam for 5*'’ grade; and 
fourth, the school district’s 2"'* and 3^'* quarter criterion referenced tests. The expected 
growth rate is a 10% decrease in students scoring below the npmi in language arts, 
reading, and mathematics on the state benchmark exam, SAT-9 exam, and the school 
district’s criterion referenced tests. 

Other Schooling Variables 

In additional to the academic achievement data, other relevant quantitative and 
qualitative data will be collected and evaluated. For example, grades, attendance. 
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discipline, constituent satisfaction, instructional strategies, charter school impact, and 
cost of instruction, administration and transportation. 

Charter School Effectiveness 

The evaluation process is used to identify needs, set priorities among needs and to 
translate needs into program objectives or facilitate the modifications of existing 
objectives (Sanders, 1992). The evaluation process is supposed to be able to gauge 
program effectiveness. In essence, the more successful a charter school is in meeting its 
assessment criteria the more effective it is deemed. The elementary school under 
discussion has a total of fourteen assessment methods. 

Singularly, none of the methods used will be able to determine whether or not the 
stated goals of the charter school are met, i.e., it is an effective early 
intervention/prevention for academic underachievement program, it is an effective early 
intervention/prevention program for socio-behavioral and personal failure, and it is an 
aggressive approach to rebuild community and parent support for public education 
through an innovative national model Collectively, the methods used will probably not 
be able to determine whether or not at-risk 3rd, 4'*’, and S'** grade students learn and 
perform significantly better than comparable students on a variety of achievement 
measures and other schooling variables (grades, attendance, discipline, constituent 
satisfaction, instructional strategies, charter school impact, and cost of instruction, 
administration and transportation) because of the relational and quantifiable problems 
associated with the assessment approach (See Table Four). For instance, what is the 
relationship between student achievement test scores and student discipline? Can the 
relationship between student performance and charter school impact on the district be 
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quantified? Even if the relationship between achievement test scores and instructional 
cost be quantified the issue of what does it mean arises. Answers to these questions and 
dozens of more indicate the difficulty associated with having an evaluation approach that 
uses so many assessment methods. 

Pandora’s Box and Charter School Evaluation 
Evaluation as a process should be focused on program needs, processes or 
strategies for providing services to learners, resource allocation, and program outcomes. 
Evaluation should be both formative and summative in scope. The evaluation approach 
should be manageable and reasonable in dimension and intent. Finally, the evaluation 
approach should pre-address the relational and quantifiable problems. 

The elementary charter school plan under discussion is typical. It is 
comprehensive to the point of perhaps being inadequate. It has a little bit of everything in 
it so that it can response to all possible concerns. As such, it contains all of the gifts in 
Pandora’s Box. It also contains the problems that arise when evaluation tries to be all 
things to all people at the same time. The central issue for evaluators of charter schools is 
to recognize this fact. 
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Table 1 



Types of Assessments 

Standardized Assessment 

Criterion Referenced Assessment 
Norm Referenced Assessment 
Performance Assessments 
Student Portfolios 

Student Demonstrations of Their Work 
Parent Satisfaction Surveys 
Student Interviews or Surveys 
Behavioral Indicators 
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Table 2 



Other Types of Assessments 

Use of Financial Resources 

Comply with State and Federal Regulations 

Impact on Student Attendance 

Instructional Practices 

Impact on Other Schools 

Cost of Instruction 

Administrative Cost 

District Related Cost 
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Table 3 



Elementary 

School Charter School Types of Assessments 
Standardized Assessment 

Criterion Referenced Assessment: School District Benchmark Tests (2"^ & 

3*^^ quarters), State Benchmark Test (4‘*' 
grade) 

Norm Referenced Assessment: Stanford Achievement Test (S*** grade) 

& Northwest Evaluation Association Level 
Tests (3*^^ -S*** grades) 

Performance Assessments 
Grades 

Parent Satisfaction Surveys 

Student Interviews or Surveys 

District Impact Surveys 

Use of Financial Resources 

Comply with State and Federal Regulations 

Impact on Student Attendance 

Impact on Student Discipline 

Instructional Practices 

Impact on Other Schools 

Cost of Instruction 

Administrative Cost 

District Related Cost: Transportation & Related Start Up Cost 
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Table 4 



Relational and Quantifiable Problems of the Charter Elementary School 

Assessment Method 

Questions to Highlight the Problems 



What is the relationship between student academic achievement and student 
discipline? Measurable? Quantifiable? 

What is the relationship between student performance and charter school impact on 
the district? Measurable? Quantifiable? 

What is the relationship between student academic achievenient and instructional 
cost? Measurable? Quantifiable? 
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